Upgrade Elasticsearch from 7.x to 8.18.0 #6640

smithellis · 2025-04-30T14:32:09Z

This commit upgrades our Elasticsearch infrastructure from version 7.x to 8.18.0,
including:

Import both elasticsearch7 and elasticsearch8
Poll the server to determine version, and use correct one
Removed elasticsearch-dsl dependency in favor of elasticsearch.dsl when on ES8
Modified ES connection settings and client configuration options
Updated ES module imports from elasticsearch_dsl to elasticsearch.dsl
Renamed test_es7_utils.py to test_es_utils.py with updated test cases
Added new ES client configuration options in settings.py
Fixed ES URL prefix to use http:// protocol explicitly
Made necessary code adjustments throughout search modules for compatibility

This upgrade provides better security, performance improvements, and access to newer
Elasticsearch features while maintaining backward compatibility with our existing
search functionality.

NOTE: We will need to plan and execute an upgrade path with hosted Elastic; this PR only solves code compatibility caused by breaking changes.

akatsoulas · 2025-05-02T10:05:39Z

kitsune/search/base.py

-            )
+
+        # Handle special case where an index with same name as alias exists
+        try:


Is there a reason for adding this check?

I think I can drop this; I added this when troubleshooting test issues with index and alias collisions. I'll see if dropping it causes problems to recur.

akatsoulas · 2025-05-02T10:09:04Z

kitsune/search/es_utils.py

+            "request_timeout": settings.ES_TIMEOUT,
+            "retry_on_timeout": settings.ES_RETRY_ON_TIMEOUT,
+            # SSL settings - these are needed for ES8 which requires SSL by default
+            "verify_certs": settings.ES_VERIFY_CERTS,


what's the reason for introducing all these new settings?

ES8 requires the SSL settings. retry_on_timeout was an attempt to improve reliability when troubleshooting test issues; the same with the sniff_on_start and sniff_on_connection_fail.
We can drop the sniff_* settings, but should consider leaving the retry_on_timeout. May not be critical but it could help when/if there are real connection issues.

akatsoulas · 2025-05-02T10:09:20Z

kitsune/search/es_utils.py

+            "sniff_on_connection_fail": settings.ES_SNIFF_ON_CONNECTION_FAIL,
+        }
+
+        if settings.ES_HTTP_AUTH:


Why is this needed?

Authentication isn't always needed, and since it seems conditional it's pulled out on its own. That separates auth stuff from the more consistent/base connection settings, which we will always have.
I think it's a good pattern but am open to feedback.
That said, I don't think we have a case where we don't use auth.

akatsoulas · 2025-05-02T10:09:50Z

kitsune/search/es_utils.py

@@ -134,10 +159,15 @@ def index_object(doc_type_name, obj_id):
        # just return
        return

+    kwargs = {}
+    # For ES8, use string "true" instead of boolean True for refresh parameter
+    if settings.TEST:


You already have this check above

There are three checks for settings.TEST in the file. The es_client has one to modify timeout and retry as part of local testing, index_object() and delete_object() both test to set the refresh to true. It didn't seem reasonable to pull that check into a helper function just for the second two tests. I don't think we want refresh set to true generally, as it forces an immediate refresh which would be expensive in prod.
I think we could remove the code I added for troubleshooting/resolving tests (so the es_client() timeout, retries and retry_on_timeout) and that would get rid of the test in es_client(). But we want the test in the other two places for sure.

akatsoulas · 2025-05-02T10:10:32Z

kitsune/search/es_utils.py

+    kwargs = {}
+    # For ES8, use string "true" instead of boolean True for refresh parameter
+    if settings.TEST:
+        kwargs["refresh"] = "true"


What's the purpose of this argument?

It forces ES to refresh, making changes immediately visible to search. So it helps for local testing more than anything else.

akatsoulas · 2025-05-02T10:15:46Z

kitsune/search/tests/signals/test_forums.py

@@ -7,6 +7,7 @@

 class ForumDocumentSignalsTests(Elastic7TestCase):
    def setUp(self):
+        super().setUp()


I am not sure that we need this

Since I changed the ElasticTestCase it is needed. Now we set the test environment up cleanly, force a refresh, and get in front of any index issues.

akatsoulas · 2025-05-02T10:16:24Z

kitsune/search/tests/test_es_utils.py

@@ -46,9 +48,16 @@ def make_first_doc_throw_exception(self, *args, **kwargs):
            question = Question.objects.get(id=question_id)
            AnswerFactory(question=question, content=f"answer {question_id}")

+        # Force ES index refresh before testing
+        QuestionDocument._index.refresh()


Why are you introducing this change? Did something change between 7 and 8?

Just more ensuring that tests don't fail because of a delay between adding something to the index and it appearing in search results. So, tests won't fail due to timing issues.

7 and 8. We import both python libraries `elasticsearch7` and `elasticsearch8` We test the server version and set it in `settings.py` We use the correct library based on the server major version Server vwrsion check is now in `search/utils.py` and uses module level caching to prevent unneeded connections

ES818

f566a09

smithellis mentioned this pull request Apr 30, 2025

Upgrade ES to 8.17 #6612

Closed

smithellis added 2 commits April 30, 2025 13:11

lint

791acce

lint some more

5b0fd1c

smithellis mentioned this pull request Apr 30, 2025

Upgrade Elastic to version 8+ mozilla/sumo#1023

Open

akatsoulas reviewed May 2, 2025

View reviewed changes

smithellis force-pushed the es818-update branch from 65d9374 to edab242 Compare May 2, 2025 18:25

remove extraneous connectivity checks

bace5bc

smithellis force-pushed the es818-update branch 3 times, most recently from 8fcee02 to bef0b8a Compare May 4, 2025 14:23

smithellis force-pushed the es818-update branch from bef0b8a to 1ffb3f4 Compare May 5, 2025 13:37

smithellis mentioned this pull request May 6, 2025

Implement ES with ES8/ES7 compatability mozilla/sumo#2323

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Upgrade Elasticsearch from 7.x to 8.18.0 #6640

Upgrade Elasticsearch from 7.x to 8.18.0 #6640

Uh oh!

smithellis commented Apr 30, 2025 •

edited

Loading

Uh oh!

akatsoulas May 2, 2025

Uh oh!

smithellis May 2, 2025

Uh oh!

akatsoulas May 2, 2025

Uh oh!

smithellis May 2, 2025

Uh oh!

akatsoulas May 2, 2025

Uh oh!

smithellis May 2, 2025

Uh oh!

akatsoulas May 2, 2025

Uh oh!

smithellis May 2, 2025

Uh oh!

akatsoulas May 2, 2025

Uh oh!

smithellis May 2, 2025

Uh oh!

akatsoulas May 2, 2025

Uh oh!

smithellis May 2, 2025

Uh oh!

akatsoulas May 2, 2025

Uh oh!

smithellis May 2, 2025

Uh oh!

Uh oh!

Upgrade Elasticsearch from 7.x to 8.18.0 #6640

Are you sure you want to change the base?

Upgrade Elasticsearch from 7.x to 8.18.0 #6640

Uh oh!

Conversation

smithellis commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

smithellis commented Apr 30, 2025 •

edited

Loading